{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 长短期记忆网络\n",
    "(Long Short-Term Memory, LSTM)\n",
    "## 6.1 基本概念\n",
    "长短期记忆网络（LSTM）是一种特殊的循环神经网络（RNN），它可以学习长期依赖性。LSTM 由 Hochreiter 和 Schmidhuber 于 1997 年提出，并被许多后续的研究者改进和推广。<br/>\n",
    "## 6.2 关键技术\n",
    "LSTM 的关键技术包括遗忘门、输入门、输出门和单元状态。<br/>\n",
    "![lstm-network](../images/6-lstm-network.webp)<br/>\n",
    "遗忘门：决定了哪些信息应该被遗忘或者抛弃。<br/>\n",
    "输入门：决定了哪些新的信息应该被存储在单元状态中。<br/>\n",
    "输出门：基于单元状态，决定了应该输出什么样的信息。<br/>\n",
    "单元状态：是 LSTM 的“记忆”部分，它在整个序列中传递信息。<br/>\n",
    "![lstm-network](../images/6-lstm-network2.webp)<br/>\n",
    "LSTM 的前向传播过程可以用以下数学公式表示：<br/>\n",
    "![lstm-network](../images/6-lstm-math.webp)<br/>\n",
    "其中， 、 和 分别是遗忘门、输入门和输出门的激活值， 是单元状态，ℎ 是隐藏状态， 和 是权重和偏置， 是 Sigmoid 函数，∗ 表示元素级别的乘法。\n",
    "## 6.3 应用领域\n",
    "LSTM 广泛应用于自然语言处理、语音识别、时间序列预测等领域。\n",
    "## 6.4 优点\n",
    "LSTM 的主要优点是可以有效地处理序列中的长期依赖问题，而且对于不同长度的序列，无需进行调整就可以进行处理。\n",
    "## 6.5 缺点\n",
    "LSTM 的主要缺点是计算复杂度高，需要大量的计算资源和时间来训练。\n",
    "## 6.6 实例分析\n",
    "LSTM 在许多自然语言处理任务中都取得了显著的成果，如机器翻译、情感分析和文本生成等。\n",
    "## 6.7 手动实现\n",
    "以下是一个简单的 LSTM 的 Python 实现："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "class LSTM:\n",
    "    def __init__(self, input_size, hidden_size, output_size):\n",
    "        self.hidden_size = hidden_size\n",
    "        self.Wf = np.random.randn(hidden_size, input_size + hidden_size)\n",
    "        self.Wi = np.random.randn(hidden_size, input_size + hidden_size)\n",
    "        self.WC = np.random.randn(hidden_size, input_size + hidden_size)\n",
    "        self.Wo = np.random.randn(hidden_size, input_size + hidden_size)\n",
    "        self.Wy = np.random.randn(output_size, hidden_size)\n",
    "        self.bf = np.zeros((hidden_size, 1))\n",
    "        self.bi = np.zeros((hidden_size, 1))\n",
    "        self.bC = np.zeros((hidden_size, 1))\n",
    "        self.bo = np.zeros((hidden_size, 1))\n",
    "        self.by = np.zeros((output_size, 1))\n",
    "\n",
    "    def forward(self, inputs):\n",
    "        h = np.zeros((self.hidden_size, 1))\n",
    "        C = np.zeros((self.hidden_size, 1))\n",
    "        ys = []\n",
    "        for i in inputs:\n",
    "            z = np.concatenate((h, i), axis=0)\n",
    "            f = sigmoid(np.dot(self.Wf, z) + self.bf)\n",
    "            i = sigmoid(np.dot(self.Wi, z) + self.bi)\n",
    "            C_bar = np.tanh(np.dot(self.WC, z) + self.bC)\n",
    "            C = f * C + i * C_bar\n",
    "            o = sigmoid(np.dot(self.Wo, z) + self.bo)\n",
    "            h = o * np.tanh(C)\n",
    "            y = softmax(np.dot(self.Wy, h) + self.by)\n",
    "            ys.append(y)\n",
    "        return ys, h, C"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这段代码首先定义了一个 LSTM 类，然后在 forward 方法中实现了 LSTM 的前向传播过程。"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
